Inter-device flow control

ABSTRACT

A network switching device comprises first and second ports. A queue communicates with the second port, stores frames for later output by the second port, and generates a congestion signal when filled above a threshold. A control module selectively sends an outgoing flow control message to the first port when the congestion signal is present, and selectively instructs the second port to assert flow control when a flow control message is received from the first port if the received flow control message designates the second port as a target.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.11/254,588, filed Oct. 20, 2005, which claims the benefit of U.S.Provisional Application Nos. 60/724,942, filed Oct. 7, 2005, 60/623,557,filed Oct. 29, 2004, and 60/679,845 filed May 11, 2005. The disclosuresof the above applications are incorporated herein by reference in theirentirety.

FIELD OF THE INVENTION

The present invention relates to buffer management in a switch.

BACKGROUND OF THE INVENTION

Referring now to FIG. 1, a schematic illustration of a four-port switch102 according to the prior art is depicted. In this example, port 2 isreceiving stream B. A stream is a series of frames having a commonsource and destination. Stream B is destined for port 4 and is shownexiting port 4. Stream A, arriving on port 3, is bound for port 1 and isshown exiting port 1. The switch 102 may be connected to network devicesthat operate at different speeds—for example 10 Mbps, 100 Mbps, or 1Gbps. If a network device connected to port 3 is operating at 1 Gbps anda device connected to port 1 is operating at 100 Mbps, port 1 may not beable to keep up with the data provided by port 3. The switch 102 may,instead of dropping frames, store frames in a queue 104. The queue 104is finite, however, and after continued transmission at 1 Gbps to port3, and only 100 Mbps out of port 1, the data will exceed the capacity ofthe queue 104 and frames will be dropped.

Once the queue 104 reaches a predetermined threshold 106, the switch 102will instruct port 3 to issue flow control (if flow control is enabled)so that the queue 104 does not overflow and force the switch 102 to dropframes. The flow control may constitute providing backpressure orsending out a MAC PAUSE. While port 3 is paused and the queue 104 isdecreasing, stream B can proceed from port 2 to port 4 withoutinterference.

SUMMARY OF THE INVENTION

A network switching device comprises first and second ports. A queuecommunicates with the second port, stores frames for later output by thesecond port, and generates a congestion signal when filled above athreshold. A control module selectively sends an outgoing flow controlmessage to the first port when the congestion signal is present, andselectively instructs the second port to assert flow control when a flowcontrol message is received from the first port if the received flowcontrol message designates the second port as a target.

In other features, the control module selectively instructs the secondport to assert flow control when the received flow control messagedesignates the switching device as a target. The control module sendsthe received flow control message to a third port when the received flowcontrol message designates a target as a device distinct from theswitching device. The threshold is dynamically set based upon a numberof free buffers within the switching device. The threshold is set to apredetermined value based upon experimental results.

In other features, the second port asserts flow control until a timerexpires. The control module selectively resets the timer to a reset timewhen a flow control message is received. The received flow controlmessage contains remote port speed information. The reset time is basedupon the remote port speed information. The flow control is asserted fora predetermined period based upon a line speed contained within thereceived flow control message. The control module selectively sends anoutgoing flow control message when a flow control enable flag is set.

In other features, at least one of a structure of the outgoing flowcontrol message is derived from the frames, a modified copy of one ofthe frames serves as the outgoing flow control message, the outgoingflow control message is based on an IEEE 802.3 frame, and the outgoingflow control message includes target device and target port information.The control module designates a target of the outgoing flow controlmessage to be a port of a device corresponding to a frame that filledthe queue above the threshold. The target of the outgoing flow controlmessage is the port of the device corresponding to the frame that filledthe queue above the threshold. The outgoing flow control messageincludes a layer three switch. The outgoing flow control messageincludes trunk information of a port of a device corresponding to aframe that filled the queue above the threshold.

In other features, the outgoing flow control message is based on an IEEE802.3 frame, and the target device and target port information is storedin a four-byte IEEE 802.3ac frame extension. The outgoing flow controlmessage is selectively set to high priority. The priority of theoutgoing flow control message is selectively set to the high prioritywhen a priority forcing flag is set. If a trunk includes the second portand a third port, the control module instructs both of the second andthird ports to assert flow control. The flow control comprises one of aMAC PAUSE and backpressure. The flow control comprises storing framesreceived by the second port in an input buffer.

In other features, a switching system comprises first and secondswitching devices. The first port of the first switching devicecommunicates with the first port of the second switching device via afirst channel. The first channel is one of a network link and aspecialized interconnection link.

A method for operating a network switching device comprises providingfirst and second ports; storing frames for later output by the secondport in a queue; generating a congestion signal when the queue is filledabove a threshold; selectively sending an outgoing flow control messageto the first port when the congestion signal is present; and selectivelyinstructing the second port to assert flow control when a flow controlmessage is received from the first port if the received flow controlmessage designates the second port as a target.

In other features, the method comprises selectively instructing thesecond port to assert flow control when the received flow controlmessage designates the switching device as a target. The methodcomprises sending the received flow control message to a third port whenthe received flow control message designates a target as a devicedistinct from the switching device. The method comprises dynamicallysetting the threshold based upon a number of free buffers within theswitching device. The method comprises setting the threshold to apredetermined value based upon experimental results. The methodcomprises providing a timer, wherein the second port asserts flowcontrol until the timer expires; and selectively resetting the timer toa reset time when a flow control message is received, wherein thereceived flow control message contains remote port speed information andwherein the reset time is based upon the remote port speed information.

In other features, the method comprises asserting the flow control for apredetermined period based upon a line speed contained within thereceived flow control message. The method comprises selectively sendingan outgoing flow control message when a flow control enable flag is set.At least one of: deriving a structure of the outgoing flow controlmessage from the frames; using a modified copy of one of the frames asthe outgoing flow control message; basing the outgoing flow controlmessage on an IEEE 802.3 frame; and including target device and targetport information in the outgoing flow control message.

In other features, the method comprises designating a target of theoutgoing flow control message to be a port of a device corresponding toa frame that filled the queue above the threshold. The target of theoutgoing flow control message is the port of the device corresponding tothe frame that filled the queue above the threshold. The outgoing flowcontrol message includes a layer three switch. The outgoing flow controlmessage includes trunk information of a port of a device correspondingto a frame that filled the queue above the threshold. The outgoing flowcontrol message is based on an IEEE 802.3 frame, and the target deviceand target port information is stored in a four-byte IEEE 802.3ac frameextension. The method comprises selectively setting the outgoing flowcontrol message to high priority wherein the priority of the outgoingflow control message is selectively set to the high priority when apriority forcing flag is set.

In other features, the method comprises providing a third port; andinstructing both of the second and third ports to assert flow control ifa trunk includes the second and third ports. The flow control comprisesone of a MAC PAUSE and backpressure. The method comprises storing framesreceived by the second port in an input buffer. The method comprisesproviding first and second switching devices, wherein the first port ofthe first switching device communicates with the first port of thesecond switching device via a first channel. The first channel is one ofa network link and a specialized interconnection link.

A network switching device comprises first and second port means forcommunicating. Queue means communicates with the second port means forstoring frames for later output by the second port means, and forgenerating a congestion signal when filled above a threshold. Controlmeans selectively sends an outgoing flow control message to the firstport means when the congestion signal is present, and selectivelyinstructs the second port means to assert flow control when a flowcontrol message is received from the first port means if the receivedflow control message designates the second port means as a target.

In other features, the control means selectively instructs the secondport means to assert flow control when the received flow control messagedesignates the switching device as a target. The control means sends thereceived flow control message to a third port means for communicatingwhen the received flow control message designates a target as a devicedistinct from the switching device. The threshold is dynamically setbased upon a number of free buffers within the switching device. Thethreshold is set to a predetermined value based upon experimentalresults. The second port means asserts flow control until timing meansfor timing expires. The control means selectively resets the timingmeans to a reset time when a flow control message is received. Thereceived flow control message contains remote port speed information.The reset time is based upon the remote port speed information. The flowcontrol is asserted for a predetermined period based upon a line speedcontained within the received flow control message. The control meansselectively sends an outgoing flow control message when a flow controlenable flag is set. At least one of: a structure of the outgoing flowcontrol message is derived from the frames; a modified copy of one ofthe frames serves as the outgoing flow control message; the outgoingflow control message is based on an IEEE 802.3 frame; and the outgoingflow control message includes target device and target port meansinformation.

In other features, the control means designates a target of the outgoingflow control message to be a port of a device corresponding to a framethat filled the queue means above the threshold. The target of theoutgoing flow control message is the port of the device corresponding tothe frame that filled the queue means above the threshold. The outgoingflow control message includes a layer three switch. The outgoing flowcontrol message includes trunk information of a port means of a devicecorresponding to a frame that filled the queue means above thethreshold. The outgoing flow control message is based on an IEEE 802.3frame, and the target device and target port means information is storedin a four-byte IEEE 802.3ac frame extension. The outgoing flow controlmessage is selectively set to high priority and wherein the priority ofthe outgoing flow control message is selectively set to the highpriority when a priority forcing flag is set.

In other features, if a trunk includes the second port means and thirdport means, the control means instructs both of the second and thirdport means to assert flow control. The flow control comprises one of aMAC PAUSE and backpressure. Input buffer means stores data for thesecond port means. The flow control comprises storing frames received bythe second port means in the input buffer means.

A switching system comprises first and second switching devices Thefirst port means of the first switching device communicates with thefirst port means of the second switching device via a first channel. Thefirst channel is one of a network link and a specialized interconnectionlink.

A computer program executable by a processor for operating a networkswitching device comprises providing first and second ports; storingframes for later output by the second port in a queue; generating acongestion signal when the queue is filled above a threshold;selectively sending an outgoing flow control message to the first portwhen the congestion signal is present; and selectively instructing thesecond port to assert flow control when a flow control message isreceived from the first port if the received flow control messagedesignates the second port as a target.

In other features, the computer program comprises selectivelyinstructing the second port to assert flow control when the receivedflow control message designates the switching device as a target. Thecomputer program comprises sending the received flow control message toa third port when the received flow control message designates a targetas a device distinct from the switching device. The computer programcomprises dynamically setting the threshold based upon a number of freebuffers within the switching device. The computer program comprisessetting the threshold to a predetermined value based upon experimentalresults. The computer program comprises providing a timer, wherein thesecond port asserts flow control until the timer expires; andselectively resetting the timer to a reset time when a flow controlmessage is received, wherein the received flow control message containsremote port speed information and wherein the reset time is based uponthe remote port speed information.

In other features, the computer program comprises asserting the flowcontrol for a predetermined period based upon a line speed containedwithin the received flow control message. The computer program comprisesselectively sending an outgoing flow control message when a flow controlenable flag is set. At least one of: deriving a structure of theoutgoing flow control message from the frames using; a modified copy ofone of the frames as the outgoing flow control message; the outgoingflow control message is based on an IEEE 802.3 frame; and includingtarget device and target port information in the outgoing flow controlmessage.

In other features, the computer program comprises designating a targetof the outgoing flow control message to be a port of a devicecorresponding to a frame that filled the queue above the threshold. Thetarget of the outgoing flow control message is the port of the devicecorresponding to the frame that filled the queue above the threshold.The outgoing flow control message includes a layer three switch. Theoutgoing flow control message includes trunk information of a port of adevice corresponding to a frame that filled the queue above thethreshold. The outgoing flow control message is based on an IEEE 802.3frame, and the target device and target port information is stored in afour-byte IEEE 802.3ac frame extension. The computer program comprisesselectively setting the outgoing flow control message to high priorityThe priority of the outgoing flow control message is selectively set tothe high priority when a priority forcing flag is set.

In other features, the computer program comprises providing a thirdport; and instructing both of the second and third ports to assert flowcontrol if a trunk includes the second and third ports. The flow controlcomprises one of a MAC PAUSE and backpressure. The computer programcomprises storing frames received by the second port in an input buffer.The computer program comprises providing first and second switchingdevices. The first port of the first switching device communicates withthe first port of the second switching device via a first channel. Thefirst channel is one of a network link and a specialized interconnectionlink.

A network switching system comprises a managing device. A firstswitching device comprises first and second ports. The first portcommunicates with the managing device. A queue communicates with thesecond port, stores frames for later output by the second port, andgenerates a congestion signal when filled above a threshold. A controlmodule directs frames received from the second port to the first port,and selectively sends an outgoing flow control message to the first portwhen the congestion signal is present.

In other features, the managing device uses the outgoing flow controlmessages to determine a rate at which to send data frames to the firstswitching device. The managing device communicates with the first portvia one of a network link and a dedicated interconnection link. At leastone of the threshold is dynamically set based upon a number of freebuffers within the first switching device and the threshold is set to apredetermined value based upon experimental results. The control moduleselectively instructs the second port to assert flow control when a flowcontrol message is received from the first port if the received flowcontrol message designates the second port as a target. The firstswitching device further comprises a timer. The second port asserts flowcontrol until the timer expires. The control module selectively resetsthe timer to a reset time when a flow control message is received,wherein the received flow control message contains remote port speedinformation. The reset time is based upon the remote port speedinformation.

In other features, the control module selectively resets the timer whenthe reset time is greater than a current value of the timer. If a trunkincludes the second port and a third port, the control module instructsboth of the second and third ports to assert flow control. The flowcontrol comprises one of a MAC PAUSE and backpressure. The firstswitching device further comprises an input buffer for the second port.The flow control comprises storing frames received by the second port inthe input buffer. The control module selectively sends an outgoing flowcontrol message when a flow control enable flag is set. At least one of:a modified copy of one of the frames serves as the outgoing flow controlmessage; the outgoing flow control message is based on an IEEE 802.3frame; the outgoing flow control message includes target device andtarget port information; the control module designates a target of theoutgoing flow control message to be a port of a device corresponding toa frame that filled the queue above the threshold; and the target of theoutgoing flow control message is the port of the device corresponding tothe frame that filled the queue above the threshold. The outgoing flowcontrol message includes trunk information of a port of a devicecorresponding to a frame that filled the queue above the threshold.

In other features, the outgoing flow control message includes a speed ofthe second port. The outgoing flow control message is based on an IEEE802.3 frame, and the speed is stored in a four-byte IEEE 802.3ac frameextension. A priority of the outgoing flow control message isselectively set to high priority. The priority of the outgoing flowcontrol message is selectively set to the high priority when a priorityforcing flag is set. The managing device includes one of a layer twoswitch and a layer three switch. The first switching device furthercomprises a third port. The control module directs frames received fromthe third port to the first port. The managing device uses the outgoingflow control messages from both the first switching device and a secondswitching device to determine rates at which to send frames to the firstand second switching devices, respectively.

A network switching system comprises managing means for managing. Firstswitching means for switching comprises first and second port means forcommunicating. The first port means communicates with the managingmeans. Queue means for storing communicates with the second port means,stores frames for later output by the second port means, and generates acongestion signal when filled above a threshold. Control means directsframes received from the second port means to the first port means, andselectively sends an outgoing flow control message to the first portmeans when the congestion signal is present.

In other features, the managing means uses the outgoing flow controlmessages to determine a rate at which to send data frames to the firstswitching means. The managing means communicates with the first portmeans via one of a network link and a dedicated interconnection link. Atleast one of the threshold is dynamically set based upon a number offree buffers within the first switching means and the threshold is setto a predetermined value based upon experimental results. The controlmeans selectively instructs the second port means to assert flow controlwhen a flow control message is received from the first port means if thereceived flow control message designates the second port means as atarget. The first switching means further comprises a timing means. Thesecond port means asserts flow control until the timing means expires.The control means selectively resets the timing means to a reset timewhen a flow control message is received. The received flow controlmessage contains remote port means speed information and wherein thereset time is based upon the remote port means speed information. Thecontrol means selectively resets the timing means when the reset time isgreater than a current value of the timing means.

In other features, if a trunk includes the second port means and thirdport means for communicating, the control means instructs both of thesecond and third port means to assert flow control. The flow controlcomprises one of a MAC PAUSE and backpressure. The first switching meansfurther comprises input buffer means for storing for the second portmeans, wherein the flow control comprises storing frames received by thesecond port means in the input buffer. The control means selectivelysends an outgoing flow control message when a flow control enable flagis set. At least one of: a modified copy of one of the frames serves asthe outgoing flow control message; the outgoing flow control message isbased on an IEEE 802.3 frame; the outgoing flow control message includestarget device and target port information; the control means designatesa target of the outgoing flow control message to be a port means of adevice corresponding to a frame that filled the queue means above thethreshold; and the target of the outgoing flow control message is theport means of the device corresponding to the frame that filled thequeue means above the threshold.

In other features, the outgoing flow control message includes trunkinformation of a port of a device corresponding to a frame that filledthe queue means above the threshold. The outgoing flow control messageincludes a speed of the second port means. The outgoing flow controlmessage is based on an IEEE 802.3 frame, and the speed is stored in afour-byte IEEE 802.3ac frame extension. A priority of the outgoing flowcontrol message is selectively set to high priority. The priority of theoutgoing flow control message is selectively set to the high prioritywhen a priority forcing flag is set. The managing means includes one ofa layer two switch and a layer three switch. The first switching meansfurther comprises third port means for communicating, and wherein thecontrol means directs frames received from the third port means to thefirst port means. The managing means uses the outgoing flow controlmessages from both the first and second switching means to determinerates at which to send frames to the first and second switching means,respectively.

A system with switching capability comprises a controlling device. Afirst switching device comprises first and second ports. The first portcommunicates with the controlling device. A control module selectivelyinstructs the second port to assert flow control when a flow controlmessage is received from the first port if the received flow controlmessage designates the second port as a target.

In other features, the controlling device sends flow control messages tothe first switching device to limit a rate of data flow to thecontrolling device. The second port asserts flow control until a timerexpires. The control module selectively resets the timer to a reset timewhen a flow control message is received. The received flow controlmessage contains remote port speed information. The reset time is basedupon the remote port speed information. The received flow controlmessage contains remote port speed information. The reset time isselected from a first table using the remote port speed information. Thecontrol module selectively resets the timer when the reset time isgreater than a current value of the timer.

In other features, the flow control is asserted for a predeterminedperiod and wherein the predetermined period is based upon at least oneof a parameter contained within the received flow control message andselected from a table based upon the parameter. If a trunk includes thesecond and third ports, the control module instructs both of the secondport and a third port to assert flow control. The flow control comprisesone of a MAC PAUSE and backpressure. The first switching device furthercomprises an input buffer for the second port. The flow controlcomprises storing frames received by the second port in the inputbuffer. The controlling device communicates with the first port via atleast one of a network link and a dedicated interconnection link. Thecontrolling module includes a central processing unit (CPU).

In other features, a queue communicates with the second port, storesframes for later output by the second port, and generates a congestionsignal when filled above a threshold. The control module selectivelysends an outgoing flow control message to the first port when thecongestion signal is present. The threshold is dynamically set basedupon a number of free buffers within the switching device. The thresholdis set to a predetermined value based upon experimental results. Thecontrol module selectively sends an outgoing flow control message when aflow control enable flag is set. The outgoing flow control message isbased on an IEEE 802.3 frame. The outgoing flow control message includesa speed of the second port. The outgoing flow control message is basedon an IEEE 802.3 frame, and the speed is stored in a four-byte IEEE802.3ac frame extension. A priority of the outgoing flow control messageis selectively set to high priority when a priority forcing flag is set.

A system with switching capability comprises controlling device meansfor controlling. First switching means for switching comprises first andsecond port means for communicating. The first port means communicateswith the controlling device means. Control means for selectivelyinstructing the second port means to assert flow control when a flowcontrol message is received from the first port means if the receivedflow control message designates the second port means as a target. Thecontrolling device means sends flow control messages to the firstswitching means to limit a rate of data flow to the controlling devicemeans.

In other features, the second port means asserts flow control until thetiming means expires. The control means selectively resets the timingmeans to a reset time when a flow control message is received. Thereceived flow control message contains remote port speed information andwherein the reset time is based upon the remote port speed information.The received flow control message contains remote port means speedinformation. The reset time is selected from the first storing meansusing the remote port speed information. The control means selectivelyresets the timing means when the reset time is greater than a currentvalue of the timing means. The flow control is asserted for apredetermined period. The predetermined period is based upon at leastone of a parameter contained within the received flow control messageand selected from a table based upon the parameter. If a trunk includesthe second port means and third port means for communicating, thecontrol means instructs both of the second and third port means toassert flow control.

In other features, the flow control comprises one of a MAC PAUSE andbackpressure. The first switching means further comprises input buffermeans for storing data for the second port means, wherein the flowcontrol comprises storing frames received by the second port means inthe input buffer. The controlling device means communicates with thefirst port means via at least one of a network link and a dedicatedinterconnection link. The controlling device means includes a centralprocessing unit (CPU). Queue means communicates with the second portmeans, stores frames for later output by the second port means, andgenerates a congestion signal when filled above a threshold. The controlmeans selectively sends an outgoing flow control message to the firstport means when the congestion signal is present.

In other features, the threshold is dynamically set based upon a numberof free buffers within the switching means. The threshold is set to apredetermined value based upon experimental results. The control meansselectively sends an outgoing flow control message when a flow controlenable flag is set. The outgoing flow control message is based on anIEEE 802.3 frame. The outgoing flow control message includes a speed ofthe second port means. The outgoing flow control message is based on anIEEE 802.3 frame, and the speed is stored in a four-byte IEEE 802.3acframe extension. A priority of the outgoing flow control message isselectively set to high priority when a priority forcing flag is set.

Further areas of applicability of the present invention will becomeapparent from the detailed description provided hereinafter. It shouldbe understood that the detailed description and specific examples, whileindicating the preferred embodiment of the invention, are intended forpurposes of illustration only and are not intended to limit the scope ofthe invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will become more fully understood from thedetailed description and the accompanying drawings, wherein:

FIG. 1 is a schematic illustration of a four-port switch according tothe prior art;

FIG. 2 is a functional block diagram demonstrating an exemplaryinterconnection of three six-port switches;

FIG. 3 is a functional block diagram of an exemplary hierarchicalinterconnection according to the principles of the present invention;

FIG. 4 is a functional block diagram of an exemplary switch controlconnection according to the principles of the present invention;

FIG. 5 is a graphical depiction of exemplary fields of transmitted flowcontrol information;

FIG. 6 is a functional block diagram of an exemplary switch according tothe principles of the present invention;

FIG. 7 is a functional block diagram of an exemplary queue controlmodule according to the principles of the present invention;

FIG. 8 is a flow chart of exemplary steps performed by the queue controlmodule of FIG. 7; and

FIG. 9 is a table of initial assumptions for determining port pausetimes.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following description of the preferred embodiments is merelyexemplary in nature and is in no way intended to limit the invention,its application, or uses. For purposes of clarity, the same referencenumbers will be used in the drawings to identify similar elements. Asused herein, the term module or device refers to an application specificintegrated circuit (ASIC), an electronic circuit, a processor (shared,dedicated, or group) and memory that execute one or more software orfirmware programs, a combinational logic circuit, and/or other suitablecomponents that provide the described functionality.

Referring now to FIG. 2, a block diagram demonstrates an exemplaryinterconnection of three six-port switches, 202, 204, and 206. Ports 5and 6 of the three switches 202, 204, and 206, are regular ports thathave been configured to be interconnection ports, forming an aggregateswitch with a greater port count. Alternately, they could be dedicatedinterconnection ports. One skilled in the art will recognize thatswitches may have greater or fewer number of ports and may beinterconnected in groups of two or more switches. In addition, a singleport, two ports, or more than two ports can be used for interconnectionpurposes. In this example, a sixth port of the first switch 202communicates with a fifth port of the second switch 204. A sixth port ofthe second switch 204 communicates with a fifth port of the third switch206. A sixth port of the third switch 206 communicates with a fifth portof the first switch 202.

In this example, stream A arrives at a first port of the second switch204 and is destined for a first port of the first switch 202. Stream Barrives at a third port of the second switch 204 and is destined for athird port of the first switch 202. Stream A′ arrives on a fourth portof the third switch 206 and is also destined for the first port of thefirst switch 202. Stream A and stream B are communicated to the sixthport of the first switch 202 by the fifth port of the second switch 204.Stream A′ is communicated to the fifth port of the first switch 202 bythe sixth port of the third switch 206.

If a network device connected to the first port of the second switch 204operates at a rate of 1 Gbps, and a network device connected to thefirst port of the first switch 202 operates at something less, such as10 Mbps or 100 Mbps, a queue 208 for the first port of the first switch202 may fill up. Once it reaches a certain threshold (which may bestatic or dynamic), the first switch 202 would traditionally use flowcontrol to prevent the sixth port of the first switch 202 from adding tothe queue 208. This in turn prevents the fifth port of the second switch204 from transmitting either stream A or stream B to the first switch202. Stream B, which may be entering the third port of the second switch204 at 100 Mbps and leaving the third port of the first switch 202 at100 Mbps, would then be blocked along with stream A. This phenomenon,where an uncongested stream is blocked because of the congestion of adifferent stream, is referred to as head-of-line blocking.

Blocking can occur for a single stream between two ports operating atthe same speed when the target port experiences downstream flow controlor collisions. Blocking can also occur when two streams, operating atthe same speed, are directed to a single port operating at that samespeed. This is demonstrated in FIG. 2 by stream A′. If streams A and A′are received at 10 Mbps, and are directed to a network device connectedto the first port of the first switch 202, also operating at 10 Mbps,the first port of the first switch 202 may receive twice as much data asit can transmit. As a result, the queue 208 will fill up. The fifth andsixth ports of the first switch 202 would then both be intermittentlyblocked to allow the queue 208 to drain, once again causing head-of-lineblocking for stream B.

It would be preferable for stream A to be blocked or buffered at thefirst port of the second switch 204, so that stream B, the uncongestedstream, can transmit at full speed. A system according to the principlesof the present invention allows stream A, arriving at the first port ofthe second switch, and stream A′, arriving at the fourth port of thethird switch, to experience flow control. Flow control may take the formof MAC PAUSE for a full duplex connection, or backpressure for a halfduplex connection. Flow control may also involve storing incoming framesin an input queue of the ingress port. This approach is most effectivewhen congestion-causing bursts are of short duration. Otherwise, theinput queue will fill quickly, and another method of flow control willhave to be employed to prevent dropping frames.

In order for a switch to issue flow control based on congestion withinanother device, the switch needs to be made aware of congestioninformation. Embedding such information in standard frames that can becommunicated between the devices using existing port interconnectionsobviates the need for additional circuitry. Flow control congestionmessages can be marked with a higher priority to ensure they will arriveas quickly as possible, ahead of standard data frames. Out-of-bandsignals could also be used to communicate flow control congestioninformation from one switch to another. This has a greater physical costin terms of chip pins or circuit board traces, and requires extra sendand receive circuitry within each switch.

If a frame from stream A increases the queue 208 past the definedthreshold, a flow control congestion message can be sent to the secondswitch 204. The second switch 204 asserts flow control on port 1, fromwhich stream A originates. Likewise, if a frame from stream A′ increasesthe queue 208 past the defined threshold, the third switch 206 will alsoreceive a flow control congestion message. In this way, flow controlacross multiple ports can be handled without any tracking circuitrywithin the queue 208 or the first switch 202. At the expense of extracircuitry, the first switch 202 can keep track of which originating portwas sending the most frames to the congested queue, and modify the flowcontrol messages accordingly.

After a port has received a flow control message (this port is referredto as the paused port), it would ideally wait until the congested queue208 drains substantially before resuming sending. However, if the pausedport waits too long and the queue 208 empties, the first port of thefirst switch 202 may temporarily have nothing to send, reducing its datarate. Controlling the duration (referred to as pause time) for a pausedport to assert flow control can be accomplished in a number of ways.

One approach is for the queue that was once congested to send acomplementary unpause flow control congestion message to ports that ithad previously paused. This requires that each queue maintain a table ofall ports that it has sent flow control congestion messages to sincebecoming congested. Allocating a static table that could contain everypossible port of every possible connected switch would occupy a largeamount of memory, most of which would never be used. Dynamicallyallocating memory to maintain this table is even more complex. Inaddition, if the switch device does not have the capacity to generatearbitrary messages, this facility would need to be added to generate theunpause flow control congestion messages.

Another approach is to send a global unpause congestion message to allother switches when a queue is no longer congested. This approach willunpause all ports, even those that have been paused by still-congestedqueues. Extra flow control traffic will result to re-pause ports thatwere erroneously unpaused by the global unpause congestion message. Inthe meantime, ports that have erroneously resumed transmitting may causequeues to overflow, and drop frames, before they can be repaused. Thisapproach also requires the ability to independently generate a frame.

A further approach would be to send an unpause congestion message thatcontains the port number and device number of the now uncongested queueto all switches. This places the burden on all paused ports to keep atable of what ports have paused them. In addition to the disadvantagesof the first approach relating to maintenance of a table, a single portmay have been paused at different times by different queues. The portwould therefore need to determine how long it should remain paused basedon the previously received flow control congestion messages from thestill-congested ports.

A final approach is to send pertinent information inside the flowcontrol message to the port that is to be paused so that the paused portitself can determine its pause time. This information may include thespeed of the congested port, how many ports are transmitting to thecongested port, and how much space remains in the queue. With thisinformation, the paused port can make an estimate of how long it shouldremain paused for, and upon the termination of this time, resumesending. If the pause time was not great enough, the queue will becomecongested once again and send another flow control congestion message.

An advantage of this open-loop approach is that the paused port will, atthe end of its pause time, resume sending and not wait indefinitely foran unpause congestion message from the congested queue. An unpausecongestion message might never be sent if the device containing thecongested queue is removed from the system or if the unpause congestionmessage is lost or corrupted between switches. Without a feedback loop,some tuning for a particular application is desirable, possibly usingbasic assumptions, such as those discussed below in relation to FIG. 9,to guide these choices.

Referring now to FIG. 3, a block diagram of an exemplary hierarchicalinterconnection according to the principles of the present invention isshown. A managing device 302 communicates with a fifth port of a firstswitch 304, which is configured to act as a multiplexer (MUX). In otherwords, the first switch 304 communicates all received data frames to themanaging device 302, and transmits frames as specified by the managingdevice 302. The managing device 302 may communicate with additionalswitches 306 configured as MUXes. In some implementations, the managingdevice 302 may be a layer three (or above) switch or a smart layer twoswitch.

In this application, flow control congestion information communicatedfrom the first switch 304 to the managing device 302 is used by themanaging device 302 not to issue flow control from one of its ports, butto moderate the amount of information being sent to the first switch304. For instance, the managing device 302 knows that a first port ofthe first switch 304 operates at 10 Mbps, and therefore sends trafficdestined for the first port of the first switch 304 at a rate of 10Mbps. However, if the first port of the first switch 304 is experiencingcollisions or downstream flow control, the full 10 Mbps rate can not beachieved, and a queue 308 will begin to fill.

When the queue 308 reaches a certain threshold, the first switch 304will then communicate flow control congestion information to themanaging device 302 to allow the managing device 302 to make appropriateadjustments. If the queue 308 is dominated by frames of a certainpriority level, or if there are separate queues for different prioritylevels, this priority information might also be communicated to themanaging device 302. When this priority information is to becommunicated, the frame should not be forced to a higher priority tocause faster delivery, as this will overwrite the original priorityinformation. Therefore, a flag to disable priority forcing may beincluded. Alternately, the original priority information can be storedelsewhere in the flow control frame.

Referring now to FIG. 4, a block diagram of an exemplary switch controlconnection according to the principles of the present invention isdepicted. A controlling module 402 communicates with a fifth port of asix-port switch 404. The controlling module 402 may desire to pause orrestrict one of the ports of the switch 404. Even if the switch 404allows the controlling module 402 to specify arbitrary frames to betransmitted out a certain port, a pause frame may be interpreted by thephysical interface of the fifth port of the switch 404 and discarded.

By sending a flow control congestion message as described in FIGS. 2 and3, the controlling module 402 causes the switch 404 to assert and/ordeassert flow control on any of its other ports. This mode requires thatthe switch 404 always respond to flow control congestion messages, evenif it is not itself generating flow control congestion messages.Therefore, a flow control disable flag for the switch 404 disables thegeneration of flow control congestion messages and does not interferewith executing flow control congestion messages from the controllingmodule 402.

Referring now to FIG. 5, a graphical depiction 502 of exemplary fieldsof transmitted flow control information is depicted. The message isdesignated as a flow control congestion message, as contrasted with adata frame. The originating switch of the frame that exceeded the queuethreshold may be included. The In_Dev field of the frame is the deviceidentifier of the switch device into which the frame originally entered.This field helps the flow control congestion message get transferred tothe original device in a case where the frame must pass through one ormore devices on its return trip. The In_Port field of the frameidentifies the source port on the In_Dev switch that received theoriginal frame. This is the port that is the target of the flow controlcongestion messages. For example, a five bit In_Dev field allows forthirty-two switch devices to be interconnected, while a five bit In_Portallows thirty-two ingress ports to be identified.

The line speed (SPD) of the congested switch port is communicated toallow determination of pause time. For example, a two bit field allowsfor common speeds such as 10 Mbps, 100 Mbps, 1 Gbps, and/or a reservedspeed. Frame priority (PRI) may be included, either as a high priorityflag to ensure fast delivery, or as data to signify that a certainpriority queue is filling up. For example, two or three bits may beused. Whether the ingress port is part of a trunk (T) (discussed below)may be communicated as a single bit.

Referring now to FIG. 6, a block diagram of an exemplary switch 602according to the principles of the present invention is depicted. Theexemplary switch 602 contains six ports 603-1, 603-2, . . . , and 603-6.Each port contains a queue controller or control module 604, whichmanages one or more queues 606. For each port, the queues 606communicate with a MAC (media access control) module 608. The queuecontrol module 604 communicates directly with the MAC module 608 to sendinformation such as flow control scheduling, etc. Each queue controlmodule 604 communicates with a switch fabric 610, which directs framesbetween the six ports.

One skilled in the art will recognize that storage space for the queues606 may be shared both between queues for a single port or betweenqueues of all ports. Additionally, there may be a single queuecontroller for all six ports. When the queue control module 604generates a flow control congestion message, the message is communicatedto the switch fabric 610, which directs it to the appropriate port.Alternately, if out-of-band signaling is used to communicate flowcontrol congestion information, the switch fabric 610 or the queuecontrol module 604 may communicate such information directly to anoutput module (not shown).

Referring now to FIG. 7, a block diagram of an exemplary queue controlmodule 702 according to the principles of the present invention isdepicted. Frames arriving at the queue control module 702 areinterpreted by a flow control congestion message detector 704. The flowcontrol congestion message detector 704 determines if the frame containsflow control congestion information or if it is a non-flow-controlframe, such as a standard data frame. Flow control congestion messagesare passed to a flow control execution block 705, while other frames arepassed to a flow control creation block 706.

Within the flow control execution block 705, a parameter extractor 708receives the flow control congestion message frame. The parameterextractor module 708 removes parameters of interest from the flowcontrol congestion message and communicates them to a pause time table710. These parameters may include the speed of the congested queue, thenumber of ports attempting to send frames to the congested queue, and/orother information. Based upon these parameters and/or internal signals,such as the local port speed, the pause time table 710 selects and/orcalculates a time for which flow control should be asserted. This timeis communicated to a comparator 712 and a timer 714. A current timevalue of the timer 714 is communicated to the comparator 712 and a flowcontrol module 716.

In one use of the invention (as in FIG. 2), if the comparator 712determines that the time from the pause time table 710 is greater thanthe current time value of the timer 714, a set signal is communicated tothe timer 714. The set signal causes the timer 714 to set itself to thetime from the pause time table 710. Using the comparator ensures thatthe port that is causing the congestion is stalled at the rate of theslowest congested port to which it is sending frames.

In another use of the invention (as in FIG. 4), regardless of thecomparison result, the set signal is asserted, causing the timer's valueto be re-loaded on every received flow control congestion message.Ignoring the comparison result allows the controlling module 402 tore-start the flow of data quickly by sending in a flow controlcongestion message that produces a zero time value from the pause timetable 710. The timer 714 decrements at a set rate—for instance, every2048 ns. While the timer's current value is non-zero, the flow controlmodule 716 instructs the corresponding MAC module to assert flowcontrol. This flow control may take the form of MAC PAUSE forfull-duplex operation or backpressure for half-duplex operation.

Within the flow control congestion message creation block 706, acongested queue diverter 738 receives non-flow-control frames. Of these,non-data frames may receive special processing. Data frames are passedto a queuing module 740. The queuing module 740 places the frame in aqueue (unless no space remains, in which case the frame may be dropped).The queuing module 740 communicates a congestion signal to an AND gate742 if the queue is filled past a certain threshold. This threshold maybe dynamic, and may vary with queue levels for other ports.

If flow control congestion messaging is enabled, the AND gate 742 alsoreceives a flow enable signal. An output of the AND gate 742 iscommunicated to the congested queue diverter 738. The output of the ANDgate 742 is an enable signal when both the flow enable and congestionsignals are present. If the congested queue diverter 738 receives theenable signal, it sends the data frame to a frame mirror 744. The framemirror 744 makes a copy of the data frame and passes it to a tagmodifier 746.

The tag modifier 746 receives internal signals, such as port speedand/or port ID of the port for this queue controller 702. The tagmodifier 746 may also receive other information, such as the number offlow control congestion messages that have been sent by the queuecontroller 702, the number of frames in the congested queue and/or thenumber of ports that have sent frames to the congested queue. The tagmodifier 746 inserts this information into the mirrored frame. Oneskilled in the art will recognize that this information can be insertedin a number of places within a frame. If the frame is an Ethernet (orIEEE 802.3) frame, the 802.3ac standard provides an extra four byteswithin the Ethernet header. These bytes may be used to transmit theparameters of interest. For further discussion, U.S. patent applicationSer. No. 10/829,866, filed Apr. 21, 2004, which is hereby incorporatedby reference in its entirety. One skilled in the art will recognize thatthis approach will also work with other frame types and networkprotocols.

The modified frame is passed to a priority module 748. If the prioritymodule receives a force enable signal, it will force the priority of themirrored frame to a level determined by a priority level signal. Forcingthe data frame to a high priority will cause it to be transmitted morequickly to the destination switch device. However, unless otherprovision is made for storing the original priority, the originalpriority information will be lost. In the exemplary application of FIG.4, priority may need to be preserved, and so the force enable flag willnot be asserted (the frame will remain unchanged through the prioritymodule 748). The output of the priority module 748 is communicated tooutput circuitry 750. The output circuitry 750 communicates flow controlmessages and standard data frames from the queuing module 740 to theswitch fabric.

Referring now to FIG. 8, a flow chart depicts exemplary steps performedby the queue control module of FIG. 7. Control starts at step 802 andtransfers to step 804. Control waits for a frame to be received in step804 and, upon receipt, transfers to step 806. In step 806, the frame ischecked to determine if the frame is destined for this switch. If not,control transfers to step 808; otherwise, control transfers to step 810.In step 808, the frame is forwarded to the destination switch, or to thenext switch in the path to reach the destination switch. The In_Devfield can be used here to pass flow control congestion messages back tothe original source device. Control then returns to step 804.

In step 810, the frame is analyzed to see if it is a flow controlcongestion message frame. If it is, control transfers to step 812;otherwise, control transfers to step 814. In step 812, congested portspeed is extracted from the frame, and control transfers to step 816. Instep 816, a delay value is selected from a table indexed by thecongested port speed. Control then transfers to step 818, where theselected delay may be compared to the current timer value of thedestination port.

If chip to chip flow control congestion messages are being used (as inFIG. 2, for example) and the selected delay is greater than the timervalue, control transfers to step 820; otherwise, control returns to step804. The selected delay may be less than the timer value if the selecteddelay is due to a flow control message from a faster port (and acorresponding shorter delay time). In step 820, the timer is set to theselected delay value, and control returns to step 804. If a controllingmodule is sending flow control congestion messages (as in FIG. 4, forexample) as determined in step 819, the test of step 818 is ignored instep 819 and control always transfers to step 820, regardless of the newdelay value from step 816. If the flow control congestion messageindicates that this port was part of a trunk, any of the trunked portscould have been the source of the congesting frame, so each port in thetrunk must be paused. To achieve this, steps 818 and 820, denoted asgroup 821, will be repeated for each trunked port.

In step 814, if the frame is tagged as non-data, control transfers tostep 822, where non-data processing is performed, and control returns tostep 804. This allows for special frames that may not, or should not, beconsidered in flow control congestion message generation. Otherwise, theframe is a data frame and control transfers to step 824. In step 824,the frame is enqueued (unless the queue is full, in which case the frameis dropped), and control continues in step 826. If flow controlcongestion message generation is enabled in step 826, control transfersto step 828; otherwise, control returns to step 804. In step 828, if thequeue is filled passed its threshold, control transfers to step 830;otherwise, control returns to step 804.

In step 830, the frame is mirrored to serve as a flow control congestionmessage, and control transfers to step 832. In step 832, the port speedbits of the flow control congestion message are set to the speed of thecongested port, and control transfers to step 834. In step 834, ifforcing priority is enabled, control transfers to step 836; otherwise,control transfers to step 838. In step 836, priority bits within theflow control congestion message are set to the given priority level, andcontrol continues in step 838. In step 838, the frame is optionallycropped. Because the frame is being used only as a carrier of flowcontrol congestion information, a maximum size frame of over 1500 bytesis unnecessary. The frame can be truncated to minimum size, as theclient data will be discarded upon arrival anyway.

Control then transfers to step 840, where the flow control congestionmessage type is marked in the frame and the frame is output to theswitch fabric pointing toward the port the congesting frame came in on,and control returns to step 804. One skilled in the art will recognizethat flow control congestion messages could be generated independentlyof a mirrored frame. In fact, the physical interface may generate flowcontrol congestion messages itself so that they do not have to wait inthe egress queue along with standard data frames.

Referring now to FIG. 9, a table of initial assumptions for determiningport pause time is presented. Again, pause time refers to the periodduring which flow control will be asserted on the switch ingress portthat is causing the congestion, and the flow control may take a formother than MAC PAUSE. In an exemplary eleven-port switch having 256total buffers distributed between its eleven ports, there areapproximately 23 buffers per port. Assuming that the pause time shouldbe long enough to allow half of them to empty yields twelve buffers asthe determining factor in pause time.

The shortest delay will determine the resolution necessary for the pausetimer. For example, a minimum size Ethernet frame is currently 672 bitslong, including the interframe gap and preamble, which at 1 Gbps takes672 ns. Multiplying this minimum time by the number of buffers for flowcontrol, 12, yields 8,064 ns. In order to allow for the estimate to beoff by at least half (4,032 ns), the resolution of the pause timer couldbe 2,048 ns, which is a multiple of a common 16 ns clock period (thenext choice, 4,096 ns, is greater than 4,032 ns, and thus too large).

The maximum pause time will likely occur with a 10 Mbps destinationport. A maximum size Ethernet frame is 12,336 bits and/or(1522+preamble+IPG). Multiplying 12,336 by 100 ns (bit time at 10 Mbps)and 12 buffers, and dividing by the pause timer resolution (2,048 ns)yields 7,228. This requires a pause time register of 13 bits (8,192).One skilled in the art will recognize that 13 bits are not absolutelynecessary, as the pause time may be stored using some form of nonlinearencoding.

A table 900 collects initial assumptions for determining pause timebased on speed of the congested port 902 and speed of the paused port904. The cause 906 assumed to most likely cause the congestiondetermines what effect 908 the pause should produce. Calculations ofpause time 910 are displayed in pause time counter units (divide by 2048ns) 912.

The first analysis concerns the assumptive cause 906 of congestion whenthe congested port speed 902 is 10 Mbps. When the paused (transmitting)port 904 is 10 Mbps, congestion could be caused by collisions ordownstream flow control at the congested port, or more than two portstransmitting to the congested port. The most likely scenario is that nomore than two source ports will be transmitting at full line speed tothe destination port. If each transmitting port is slowed by half, therates will match. A 2 to 1 reduction should allow for the congested portto become uncongested.

When the paused port 904 is 100 Mbps, the most likely cause ofcongestion is simply the line speed difference. Therefore, a 10 to 1reduction should allow the queue to become uncongested. Similarly, apaused port 904 operating at 1 Gbps will likely cause congestion becauseof its hundredfold speed advantage. A 100 to 1 reduction shouldtherefore be adequate.

The second analysis concerns the assumptive cause 906 of congestion whenthe congested port speed 902 is 100 Mbps. If the paused port isoperating at only 10 Mbps, the congested port must be experiencingcollisions or flow control, or there are ports other than the 10 Mbpspaused port contributing to the traffic. Because the 10 Mbps is tentimes slower than the congested port, a reduction of 1.1 to 1 should besufficient.

When both ports are operating at 100 Mbps, collisions or more than twoports transmitting to the congested port is the cause of the congestion.It is less likely for more than two ports to be transmitting at fullspeed to a single port, so a 2 to 1 reduction will likely beappropriate. With a 1 Gbps paused port, tenfold speed disparity is thelikely cause of the congestion, and a 10 to 1 reduction will be applied.

The final analysis concerns the assumptive cause 906 of congestion whenthe congested port speed 902 is 1 Gbps. When the paused port is only 10Mbps, there must be at least two ports transmitting to the congestedport. Because the 10 Mbps port is 1/100 the speed of the congested port,a 1.01 to 1 reduction will likely be sufficient. Similarly, for a 100Mbps paused port, a 1.1 to 1 reduction should be sufficient. Finally,with a 1 Gbps paused port, at least two ports must be transmitting tothe congested port. Taking the most common scenario of two portstransmitting, a 2 to 1 reduction will be employed.

Table 900 demonstrates that pause times group by the speed of thecongested port 902. The speed of the paused port 904 is relativelyinsignificant. It may therefore be possible to determine three pausetimes based solely upon the speed of the congested port 902. As pausetimes are increased, the number of frames dropped within the switchdevices is minimized, even if many ports transmit at full line speed tothe same congested output port. High pause times, however, can lead toports pausing so long that the originally congested queue empties, andthe port is left with nothing to transmit. This reduction in the speedof the port should be avoided.

If precise traffic patterns are known in advance, the pause times can betailored so that frames are never dropped and all ports operate at fullspeed. In the majority of circumstances, traffic patterns are variable,and a compromise must be reached that keeps the pause times small enoughto prevent a decrease in port speed, while keeping the times long enoughto prevent frames being dropped under most traffic conditions. Undercertain extreme traffic conditions (which are usually brief), thiscompromise will lead to dropped frames.

An experiment began with 401, 41, and 5 timer counts for congested portspeeds of 10 Mbps, 100 Mbps, and 1 Gbps, respectively. Experimentationshowed that optimal pause times did cluster together based uponcongested port speed—i.e., the speed of the paused port had littleeffect. However, the size of transmitted frames was found to affect theoptimal pause times. When many of the frames transmitted are larger thanminimum size, the pause times above were found to be too low due to thechange in buffer efficiency and the time it took to drain the frames.The numbers presented here assume a fixed allocation of about 23 buffersper port where approximately half full is used as the congestionthreshold. The number of available buffers and how the threshold ofcongestion is determined will affect the delay times.

The open loop flow control messaging system is designed to cover themajority of typical network congestion cases with minimal system cost.But not all cases can be covered with such a simple solution. Ifextraordinary traffic patterns occur, the original IEEE PAUSE link-basedflow control mechanism can be used. In one embodiment, two thresholdsare set for each output queue. A lower threshold determines when togenerate the mirrored flow control congestion messages discussed above,and a higher threshold determines when to generate standard IEEE linkPAUSE frames on the inter-switch links (ports 5 and 6 in FIG. 2).

For example, if 100 Mbps ports 1, 2, and 4 in Switch 2 send frames tothe 10 Mbps Switch 1 Port 1 (Stream A), the flow control congestionmessage delay time may not be large enough. The Switch 1 Port 1 bufferswill fill three times faster (from the three 100 Mbps ports) but drainat the same 10 Mbps rate. The delay time is often calibrated to handleup to two streams. More steams can be supported if there is morebuffering available, or if Switch 1 Port 1 is allowed to drop below 10Mbps (the delay time can be set high enough to cause the buffers tocompletely drain).

In this case, the output queue for Switch 1 Port 1 will continue fillingas the delay times expire and will eventually pass a second, higher,threshold. At this point, a standard IEEE link PAUSE can be used to stopall traffic coming in Port 6 on Switch 1. When this occurs, Stream Bwill also be blocked until the output queue of Switch 1 Port 1 drainsenough to release the IEEE PAUSE on the link. This fall-back mechanismprevents frame loss at the expense of some head of line blocking duringpeak congestion situations. Real networks generally have only momentarycongestion and the flow control congestion message system handles thesecases without any head of line blocking.

Those skilled in the art can now appreciate from the foregoingdescription that the broad teachings of the present invention can beimplemented in a variety of forms. Therefore, while this invention hasbeen described in connection with particular examples thereof, the truescope of the invention should not be so limited since othermodifications will become apparent to the skilled practitioner upon astudy of the drawings, the specification and the following claims.

1. A network switch comprising: a first port configured to (i) receivefirst frames transmitted to the network switch from a second switch and(ii) transmit a first flow control message to the second switch; asecond port configured to (i) receive the first frames from the firstport and (ii) receive second frames transmitted to the network switch; afirst queue in communication with the second port, the first queueconfigured to (i) store the first frames to be output by the secondport, (ii) generate a congestion signal when the first queue is filledabove a threshold; and (iii) store the second frames to be output by thefirst port based on a second flow control message; and a first controlmodule configured to (i) send the first flow control message to thefirst port based on the congestion signal and (ii) instruct the secondport to assert flow control based on the second flow control message,wherein the first control module instructs the second port to assert theflow control when (i) the second flow control message is received by thefirst port and from the second switch and (ii) the second flow controlmessage designates the second port as a target device to assert the flowcontrol, and wherein the asserting of the flow control includes storingthe second frames in the first queue.
 2. The network switch of claim 1,the first port is configured to receive third frames from a third portof the network switch when the second port is asserting the flowcontrol.
 3. The network switch of claim 1, further comprising a thirdport, wherein the first control module is configured to send the secondflow control message to the third port when the second flow controlmessage designates a target as a device downstream from the networkswitch.
 4. The network switch of claim 1, further comprising a timer,wherein: the second port is configured to assert the flow control untilthe timer expires; the first control module is configured to reset thetimer to a reset time when the second flow control message is receivedby the first port; the second flow control message includes port speedinformation of a device other than the network switch; and the resettime is based on the port speed information.
 5. The network switch ofclaim 1, wherein: the second port asserts the flow control for apredetermined period; the predetermined period is based on a line speed;and the second flow control message includes a line speed.
 6. Thenetwork switch of claim 1, wherein: the first frames fill the firstqueue above the threshold; the first control module designates an inputport of the second switch to receive the first flow control message; andthe input port receives the first frames prior to the first frames beingtransmitted to the network switch via an output port of the secondswitch.
 7. The network switch of claim 1, wherein: the first portreceives the first frames at a first rate; and the first control moduleis configured to send the first flow control message to the secondswitch to adjust transmission rate of the first frames to a second ratethat is (i) greater than zero and (ii) less than the first rate.
 8. Thenetwork switch of claim 1, wherein: the first flow control messagecomprises a speed of the first queue; the first control module isconfigured to (i) select a delay based on the speed of the first queue,(ii) adjust a timer based on the delay, and (iii) generate the firstflow control message based on the timer; the network switch receives theframes from the second switch at a transmission rate; and thetransmission rate is based on the timer.
 9. The network switch of claim1, further comprising a second queue configured to receive the secondframes from the first queue, wherein: the first queue is connectedbetween the first control module and the second port, and the secondqueue is connected between the first control module and the first port.10. The network switch of claim 1, further comprising a third port,wherein the first control module sends the second flow control messageto the second port to limit transmitting of the second frames from thesecond port to the first port while permitting passage of data from thethird port to the first port.
 11. The network switch of claim 1, furthercomprising a third port, wherein the first control module is configuredto send the first flow control message to the first port to limittransmitting of the first frames to the first port while permittingpassage of data from the third port to the queue.
 12. The network switchof claim 1, further comprising: a second queue configured to store datareceived by the first port prior to transferring the data to the firstqueue; and a second control module configured to adjust a transfer rateof the data from the second queue to the first queue based on the firstflow control message.
 13. The network switch of claim 12, furthercomprising a switch fabric connected between the first control moduleand the second control module, wherein: the first control module isconfigured to determine whether a frame received from the switch fabricis one of a flow control message and a non-flow control message; and thefirst control module is configured to direct the frame received from theswitch fabric to the second control module when the frame received fromthe switch fabric is a non-flow control message.
 14. The network switchof claim 1, wherein the first control module is configured to direct acopy of the first frames received by the second port back to the firstport when the first frames received by the second port are non-flowcontrol frames.
 15. The network switch of claim 14, wherein: the firstcontrol module is configured to modify the copy to include a tagmodifier; and the tag modifier comprises at least one of a port speed ofthe second port, a port identification of the second port, a first valueindicating a quantity of flow control messages transmitted by the secondport, a second value indicating a quantity of frames stored in the firstqueue, or a third value indicating a quantity of ports, wherein thequantity of ports identifies a number of ports that have transmittedframes to the second port.
 16. The network switch of claim 14, wherein:the first control module is configured to modify the copy to include atag modifier; and the tag modifier comprises a port speed of the secondport, a port identification of the second port, a first value indicatinga quantity of flow control messages transmitted by the second port, asecond value indicating a quantity of frames stored in the first queue,and a third value indicating a quantity of ports, wherein the quantityof ports identifies a number of ports that have transmitted frames tothe second port.
 17. A system comprising: the network switch of claim 2;the second switch; and one of a managing device and a controlling devicethat is distinct from each of the network switch and the second switch.18. The system of claim 17, wherein: the system comprises the managingdevice; and the managing device is configured to adjust a rate at whichthe first frames are transmitted from the second switch to the networkswitch based on the first flow control message.
 19. The system of claim18, wherein: the first frames are received by the second switch; thesecond switch comprises a second control module; the second controlmodule directs the first frames to the managing device; and the managingdevice directs the first frames to the network switch.
 20. The system ofclaim 17, wherein: the system includes the controlling device; thecontrolling device sends a third flow control message to the first portof the network switch to limit a rate of data flow from the networkswitch to the controlling device; and the third flow control messageidentifies the second port as a target device to assert flow control.